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ABSTRACT 

The  purpose  of  the  Research  and  Development  Testbed  (R&D  Testbed)  at  the  Defense  Threat  Reduction  Agency's 
(DTRA)  Center  for  Monitoring  Research  (CMR)  is  to  improve  nuclear  explosion  monitoring  capability  by 
supporting  the  R&D  community  with  a  wide  range  of  activities  and  resources  and  providing  environments  for 
testing  and  evaluating  promising  research  results  at  a  wide  range  of  scales.  For  certain  deliverables,  DTRA  directs 
its  sponsored  researchers  to  deliver  their  research  results  to  the  R&D  Testbed.  Results  delivered  to  the  R&D 
Testbed  are  permanently  archived  in  a  manner  that  facilitates  access  by  other  researchers.  In  addition,  the  R&D 
Testbed  provides  a  facility  to  evaluate  such  research  results  and,  if  desired,  integrate  them  into  other  software 
systems  and  subsystems.  For  example  deliverables  are  both  integrated  into  software  for  delivery  to  the 
Comprehensive  Test  Ban  Treaty  Organization’s  International  Data  Centre  and  also  transitioned  to  the  US  Air  Force 
Technical  Applications  Center  (AFTAC)  at  Patrick  Air  Force  Base  in  Florida  for  routine  worldwide  nuclear 
explosion  test  monitoring.  The  purpose  of  this  paper  is  to  formally  document  the  established  procedures  of  the 
DTRA  CMR  R&D  Testbed. 

A  key  component  for  success  of  the  R&D  Testbed  is  the  delivery,  test,  and  acceptance  of  results  from  the  R&D 
community.  In  general,  the  R&D  Testbed  expects  four  types  of  deliveries:  technical  reports,  data  to  receive  and 
store,  software  components  to  evaluate  and  possibly  integrate,  and  parametric  results  (such  as  Source  Specific 
Station  Corrections)  to  evaluate  and  integrate.  Separate  procedures  are  required  for  the  different  types  of  deliveries: 

•  Technical  reports  only  need  to  be  delivered  to  the  R&D  Testbed,  preferably  electronically. 

•  Data  deliveries: 

o  Data  must  be  delivered  in  electronic  form,  by  FTP  or  tape  submission, 
o  A  description  of  the  data  format  will  aid  the  delivery;  the  preferred  format  is  CSS  3.0. 

■  Other  formats  require  discussion  to  ensure  the  format  is  properly  defined. 

•  Deliveries  that  are  to  be  evaluated  and/or  integrated  (software  components  and  parametric  results): 

o  Optimally,  a  delivery  schedule  is  defined  early  in  a  research  contract, 
o  The  R&D  Testbed  staff  can  assist  the  researcher  in  defining  the  software  interfaces, 

o  A  test  plan  with  evaluation  criteria  is  generated  by  the  researcher  in  collaboration  with  R&D 

Testbed  staff. 

o  The  component  to  be  evaluated  is  delivered, 
o  The  component  is  evaluated  at  the  R&D  Testbed. 

o  The  R&D  Testbed  provides  feedback  to  researcher  regarding  the  testing, 
o  A  test  report  is  generated  by  the  R&D  Testbed  with  collaboration  from  the  researcher, 
o  DTRA  will  direct  the  subsequent  activity:  the  component  may  be  developed  further  or  integrated 
into  the  operational  test  and  evaluation  system  at  CMR. 

The  procedure  for  delivery  and  evaluation  of  data  and  components  requires  coordination  between  the  R&D  Testbed 
and  the  sponsored  researcher.  For  data  deliveries,  the  primary  issue  is  ensuring  that  the  R&D  Testbed  is  able  to 
integrate  the  delivery  into  its  archive  database.  For  deliveries  that  are  to  be  evaluated  and/or  integrated,  the  process 
is  more  detailed  and  requires  contact  early  in  the  research  and  development  cycle  and  continuing  beyond  the 
component  delivery.  Historically,  the  duration  of  the  test  and  integration  cycle  has  been  3-6  months,  and  after 
product  delivery  has  required  significant  effort  by  both  the  R&D  Testbed  staff  and  the  researcher. 

As  of  lune  2000,  the  R&D  Testbed  expects  deliveries  over  the  next  3  years  from:  23  projects  that  require  evaluation, 
36  that  require  only  archiving  of  final  reports,  and  32  that  require  integration  of  data.  Given  the  resources  required 
to  properly  evaluate  deliveries,  every  effort  should  be  made  to  define  a  realistic  delivery  and  testing  schedule  that 
works  for  both  the  researcher  and  the  R&D  Testbed. 
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OBJECTIVE 


One  of  the  primary  objectives  of  the  Research  and  Development  Testbed  at  the  DTRA  Center  for  Monitoring 
Research  is  to  provide  a  central  delivery  point  for  test,  evaluation  and  storage  of  the  results  of  DTRA  sponsored 
Nuclear  Monitoring  research  and  development.  The  focus  of  this  paper  is  to  outline  the  procedures  established  for 
interacting  with  the  R&D  Testbed  for  the  purposes  of  accepting,  integrating,  testing,  and  evaluating  research  results. 

RESEARCH  AND  DEVELOPMENT  TESTBED  PROCEDURES 


The  R&D  Testbed  was  designed  to  handle  deliveries  ranging  in  complexity  from  technical  reports  to  software 
components  that  are  to  be  evaluated  for  potential  use  in  an  integrated  operational  environment.  Procedures  have 
been  developed  and  refined  over  time  to  facilitate  delivery  and  testing  of  various  types  of  deliverables,  where  the 
level  of  procedure  is  dependent  on  the  type  of  delivery.  For  delivery  of  reports,  all  that  is  required  is  an  electronic 
copy  of  the  report  in  a  known  format.  Other  types  of  delivery  require  a  delivery  plan,  which  includes  descriptions  of 
what  is  to  be  delivered,  its  format,  how  it  is  to  be  installed,  and  a  schedule.  In  addition,  delivery  of  items  that  are  to 
be  evaluated  must  include  a  test  plan  that  describes  suggested  test  procedures  and  expected  results.  Templates  for 
delivery  and  test  plans  are  being  published  on  the  R&D  Testbed  web  site  (http://www.cmr.gov/rdtb). 


Research  Reports 

Delivery  of  final  research  reports  requires  minimal  procedure.  One  notifies  the  R&D  Testbed  regarding  the 
delivery,  specifying  a  proposed  delivery  method  and  product  format.  The  preferred  delivery  method  is  electronic 
distribution,  with  the  format  specified  (e.g.,  MS  Word,  Frame  5.5,  PDF.). 

Once  the  report  is  delivered,  the  report  will  (with  DTRA  approval)  be  made  available  via  the  R&D  Testbed  web 
page  or  ftp  site.  In  addition,  a  hardcopy  of  the  report  will  be  archived  at  CMR. 


Table  1  Actions  for  Report  Deliveries 


Contributor 

R&D  TB 

DTRA 

1.  Electronic  delivery 
formatted  in: 

•  PDF 

•  Word 

•  FM 

•  Postscript 

1 .  Convert  to  PDF 

2.  Post  on  R&D  Testbed  web  site 
as  PDF 

3.  Reply  to  sender 

1 .  Acceptance  of 
deliverable 

2.  Clearance  as  needed  — 
prior  to  posting 

3.  Receive  copy  of  report 

Figure  1  :  The  process  flow  for  delivering  reports: 


Data  and  Parametric  Products 


Some  deliveries  to  the  R&D  Testbed  can  be  described  as  ‘Receive  and  Store’  deliveries.  That  is,  such  items  are 
received  by  the  R&D  Testbed;  they  are  subjected  to  basic  acceptance  tests,  and  are  then  stored  with  the  intent  of 
making  them  available  to  other  researchers.  Previous  examples  of  such  deliveries  would  be  waveform  data  sets, 
earthquake  catalogs,  velocity  models,  and  so  forth. 

Receive  and  Store  deliveries  to  the  R&D  Testbed  require  special  preparation  to  ensure  that  the  R&D  Testbed  will  be 
able  to  interpret  formats,  have  sufficient  metadata  to  enable  the  product’s  use,  and  be  ready  to  successfully  integrate 
the  data  set  into  the  R&D  Testbed  databases  and  storage  system.  There  are  a  number  of  general  actions  by  the 
contributor,  the  R&D  Testbed,  and  DTRA,  which  apply  to  any  product  which  will  be  stored  on  the  R&D  Testbed 
and  which  will  subsequently  be  used  by  others.  Such  actions  are  outlined  in  Table  2  below,  with  specific  examples 
provided  for  waveform  and  event  catalog  deliveries.  In  addition.  Figure  2  illustrates  the  process  flow. 


Table  2  :  Actions  for  Data  Deliveries 


Contributor 

R&D  TB 

DTRA 

Generic 

1 .  Abbreviated  delivery  plan 

1.  Verify  readability 

1 .  Acceptance  of 

Receive 

1.1.  Type  of  information  to  store 

2.  Store  as  delivered 

deliverable 

and  Store 

1.2.  Expected  delivery  size 

3.  Delivery  report 

2.  Clearance  as 

1.3.  Limits  on  redistribution 

needed  —  prior  to 

(DTRA  may  overrule) 

posting 

1 .4.  Expected  delivery  date 

3.  Assessment  of 

2.  Electronic  delivery  in  documented 

future  action  on 

format 

deliverable 

Example  1; 

1 .  Abbreviated  delivery  plan 

1 .  Review  delivery  plan 

4.  Resolution  of 

Waveform 

1.1.  Expected  delivery  size 

2.  Verify  readability 

disputes 

data 

1.1.1.  Bytes  of  data 

3.  Load  into  R&D  Testbed 

1.1.2.  Number  of  waveform 

database 

segments 

3.1.  Convert  to  CSS  3.0  format 

1.2.  Delivered  format 

3.2.  Add  any  new  stations  to 

1.2.1.  CSS  3.0  is  preferred 

R&D  Testbed  station 

1.2.2.  GSE,  SEED,  SAC,  or  AH 

listing 

acceptable 

3.3.  Load  waveform  onto  mass 

1.3.  Description  of  supplementary 

storage  device 

data  (origins,  arrivals,  etc.) 

3.4.  Load  waveform 

1 .4.  Limits  on  redistribution 

description  and  associated 

1.5.  Expected  delivery  date 

data  into  the  R&D  Testbed 

2.  Electronic  delivery  of  waveforms  in 

database 

expected  format 

4.  Prepare  delivery  report  and 

submit  to  DTRA 

5.  Post  for  export 

1 .  Abbreviated  delivery  plan 

1 .  Review  delivery  plan 

J_jA.CULllL/lt  Z.. 

1.1.  Expected  delivery  size 

2.  Verify  readability 

Catalog 

1.1.1.  Bytes  of  data 

3.  Load  into  R&D  Testbed 

1.1.2.  Number  of  events 

database 

1.2.  Delivered  format 

3.1.  Convert  to  CSS  3.0  format 

1.2.1.  CSS  3.0  is  preferred 

3.2.  Load  origin,  event,  arrival. 

1.2.2.  Any  format  with 

etc.  tables  to  R&D 

metadata  is  acceptable 

Testbed  database 

1.3.  Limits  on  redistribution 

4.  Prepare  delivery  report  and 

1 .4.  Expected  delivery  date 

submit  to  DTRA 

2.  Catalogs  in  ASCII  (DB  ready)  and 

5.  Post  for  export 

including  metadata 

Parse  int 
database 


Software  to  be  evaluated 


Procedures  for  delivering  software  to  the  R&D  Testbed  are  similar  to  that  required  for  data  deliveries.  The  primary 
differences  involve  the  need  for  additional  information  in  the  delivery  plan  such  as  configuration  issues  (e.g.,  a  list 
of  files  required  to  execute  the  code),  and  hardware  and  software  requirements  (e.g.,  the  type  of  computer  system 
required  by  the  component  and  the  expected  version  of  the  operating  system).  Of  particular  importance  to  the 
process  are  steps  ensuring  that  the  delivered  component  can  run  on  the  R&D  Testbed,  that  the  R&D  Testbed  staff 
understand  what  is  required  to  install  and  evaluate  the  component,  and  that  the  delivery  timing  allows  for  sufficient 
evaluation  of  the  component  during  the  performance  period  of  the  contributor’s  contract.  To  mitigate  potential 
issues,  a  delivery  plan  and  a  stand-alone  test  plan  are  to  be  submitted  to  the  R&D  Testbed  by  the  contributor.  In 
addition,  the  R&D  Testbed  can  supply  a  compatibility  list  of  software,  hardware,  and  commercial  off  the  shelf 
(COTS)  products. 

It  is  important  that  a  realistic  schedule,  mutually  agreeable  to  the  contributor  and  the  R&D  Testbed,  be  negotiated 
with  R&D  Testbed  staff  as  soon  as  possible  in  the  overall  process.  The  schedule  should  contain,  at  minimum,  the 
expected  date  of  submission  for  the  delivery  and  test  plans,  the  expected  delivery  date  of  the  component,  the 
evaluation  period,  and  a  final  report  date.  The  schedule  is  critical  because  any  deviation  in  the  schedule  (by  either 
party)  may  result  in  scheduling  conflicts  with  other  submissions  to  the  R&D  Testbed. 

The  stand-alone  test  plan  focuses  on  the  procedures  for  testing,  the  test  criteria,  and  test  cases.  Researchers  shall 
submit  a  test  plan  that  provides  comprehensive  information  on  testing  the  product.  This  shall  include  features  to  be 
tested,  pass/fail  criteria,  testing  approach,  resource  requirements,  and  schedules.  Test  plans  should  clearly  detail  all 
procedures  for  test  cases.  Delivery  and  Test  Plan  templates  are  available  from  the  R&D  Testbed  web  site. 

Specifications  for  each  party  are  outlined  in  Table  3  below.  In  addition.  Figure  3  on  the  following  page  illustrates 
the  process  for  delivery  and  evaluating  software  components. 

Table  3  :  Actions  for  Software  Deliveries  to  be  Evaluated 

Contributor  R&D  TB  DTRA 

1.  Prepare  delivery  plan  1.  Maintain  COTS  list  1.  Acceptance  of  deliverable 

1.1.  System  impact  2.  Supply  metadata  guidelines  2.  Clearance  as  needed  — 

1.1.1.  What  does  the  component  do?  3 .  Review  delivery  and  test  plan  prior  to  posting 

1.1.2.  What  are  the  inputs  and  outputs?  4.  Install  and  test  software  in  stand-  3.  Assessment  of  future 

1.1.3.  How  is  the  software  started?  alone  mode  action  on  deliverable 

1.2.  System  requirements  5.  Prepare  delivery  report  and  4.  Resolution  of  disputes 

1.2.1.  Hardware  needed  by  software  submit  to  DTRA 

1.2.2.  Software  requirements  (deviations  6.  Evaluate  parametric  information 

from  the  COTS  list)  7.  Prepare  evaluation  report  and 

1.3.  Configuration  issues  submit  to  DTRA 

1.4.  Support  requirements:  what  R&D 
Testbed  staff  support  will  be  required  for 
delivery 

1.5.  Delivery  schedule 

1.6.  Installation  instructions 

2.  Prepare  stand-alone  test  plan 

2.1.  Procedures  to  execute  test 

2.2.  Test  cases 

2.3.  Evaluation  criteria 

2.4.  Sample  results 

3.  Deliver  software 

3.1.  Complete  source  code 

3.2.  Consistent  with  COTS  list 

3.3.  Documentation 


Components  to  be  Evaluated  in  the  Context  of  a  Full-Scale  Monitoring  System 

Components  that  are  to  be  tested  and  evaluated  after  integration  into  a  full-scale  monitoring  system  environment,  for 
example  the  CTBT  Monitoring  Software  System  (CMSS),  require  additional  steps  than  components  that  are  to  be 
evaluated  in  a  stand-alone  environment. 

The  delivered  components  must  be  compatible  with  the  applications  in  the  full-scale  test  environment.  For  example, 
while  R&D  Testbed  facilities  include  Intel-based  machines,  the  full-scale  environment  uses  the  SUN  SPARC  family 
processors.  Special  attention  must  be  placed  on  how  the  component  is  to  be  integrated;  that  is,  what  process  will 
initiate  the  component,  where  the  component  retrieves  data,  and  where  it  writes  output.  As  such,  an  integration  plan 
is  added  to  the  delivery  requirements.  In  addition,  developers  of  these  components  are  encouraged  to  work  with  the 
R&D  Testbed  software  and  systems  engineers  as  early  as  possible  in  their  development  process  to  ensure 
compatibility  with  the  environment  of  the  R&D  Testbed.  These  additional  requirements  have  significant 
implications  regarding  the  delivery  schedule.  The  test  and  evaluation  schedule  is  tighter  for  full-scale  testing,  due  to 
competition  for  access  to  the  integrated  environment. 

An  integration  test  plan  is  required  for  components  which  will  be  integrated  and  evaluated  in  the  context  of  a  larger 
system.  Integration  tests  differ  from  the  stand-alone  tests  in  that  the  stand-alone  test  is  primarily  focused  on  the 
function  of  the  delivered  application,  whereas  integration  focuses  on  the  operation  of  the  system  as  a  whole  with  the 
delivered  component  installed.  As  such,  the  Integration  Test  Plan  provides  the  procedures  for  evaluating  the 
delivered  component  within  the  context  of  the  full-scale  test  environment.  The  primary  issues  in  the  Integration  Test 
Plan  are  the  instructions  for  determining  and  assessing  correct  interaction  between  the  system  and  the  specific 
application,  and  the  basis  for  estimating  overall  system  impact.  The  performance  of  the  application  and  the 
integrated  system,  in  terms  of  monitoring  capability  criteria,  are  assessed  in  the  context  of  the  final  system  output. 
For  example,  improvements  in  event  location  precision  and  accuracy  would  be  performance  criteria  for  location 
algorithms  or  location  calibration  parameters.  Templates  for  both  the  Integration  Plan  and  the  Integration  Test  Plan 
are  available  on  the  R&D  Testbed  web  site. 

There  are  specific  actions  for  the  contributor,  the  R&D  Testbed,  and  DTRA,  as  outlined  in  Table  4  below.  In 
addition.  Figure  4  illustrates  the  process  for  delivery  and  evaluating  software  components. 


Table  4  :  Actions  for  Software  Deliveries  to  be  Integrated 


Contributor 

R&D  TB 

DTRA 

1.  Delivery  plan  [See  Table  3] 

1. 

Maintain  COTS  list 

1 .  Acceptance  of 

2.  Integration  plan 

2. 

Supply  metadata  guidelines 

deliverable 

2.1.  List  the  application  that 

3. 

Review  delivery  and  test 

2.  Clearance  as  needed  — 

initializes  the  component 

plans 

prior  to  posting 

2.2.  Explain  all  interfaces 

4. 

Install  and  test  software  in 

3.  Assessment  of  future 

2.2.1.  Input  interfaces 

stand  alone  mode 

action  on  deliverable 

2.2.2.  Component  output 

5. 

Integrate  component 

4.  Resolution  of  disputes 

2.2.3.  Controlling  the 

according  the  integration 

component 

plan 

2.3.  Integration  instructions 

6. 

Prepare  delivery  report  and 

3.  Stand-alone  test  plan  [See  Table  3] 

submit  to  DTRA 

4.  Integration  test  plan 

7. 

Prepare  evaluation  plan  and 

4.1.  Procedures  to  execute  test 

submit  to  DTRA 

4.2.  Evaluation  methodology 

8. 

Evaluate  parametric 

4.3.  Expected  system  impact 

information 

5.  Deliver  software 

9. 

Prepare  evaluation  report 

5.1.  Complete  source  code 

and  submit  to  DTRA 

5.2.  Consistent  with  COTS  list 

5.3.  Documentation 

SUMMARY 


The  DTRA  CMR  Research  and  Development  Testbed  is  a  designated  testing  and  archival  delivery  point  for  DTRA 
sponsored  research  efforts  The  R&D  Testbed  has  the  capability  to  test  and  evaluate  deliveries  in  stand-alone  mode, 
and  in  the  context  of  the  CTBT  Monitoring  System  Software.  Because  of  the  complexity  of  receiving,  storing, 
distributing,  and  evaluating  a  wide  range  of  research  products,  a  basic  level  of  procedure  is  essential  and  is 
proportional  to  the  complexity  of  the  delivery: 

1 .  Research  report  deliveries  require: 

•  Using  an  acceptable  electronic  format 

2.  Deliveries  of  data  and  parametric  information  require: 

•  Prepare  an  abbreviated  delivery  plan 

3.  Deliveries  of  stand-alone  components  require: 

•  A  delivery  plan 

•  A  stand-alone -test  plan 

4.  Deliveries  of  components,  information,  or  systems  intended  for  integration  require: 

•  A  delivery  plan 

•  An  integration  plan 

•  A  stand-alone  test  plan 

•  An  integration  test  plan 


